Understanding the Causes of Consistency Anomalies in Apache Cassandra

A recent paper on benchmarking eventual consistency showed that when a constant workload is applied against Cassandra, the staleness of values returned by read operations exhibits interesting but unexplained variations when plotted against time. In this paper we reproduce this phenomenon and investigate in greater depth the low-level mechanisms that give rise to stale reads. We show that the staleness spikes exhibited by Cassandra are strongly correlated with garbage collection, particularly the “stop-the-world” phase which pauses all application threads in a Java virtual machine. We show experimentally that the staleness spikes can be virtually eliminated by delaying read operations artificially at servers immediately after a garbage collection pause. In our experiments this yields more than a 98% reduction in the number of consistency anomalies that exceed 5ms, and has negligible impact on throughput and latency. 1. OBJECTIVE AND CONTRIBUTIONS In this paper, we study the staleness of values returned by read operations applied to Cassandra—an open-source distributed storage system that supports eventual consistency using quorum-based replication. More precisely, we investigate the causes of sharp spikes that occur when a constant workload is applied to Cassandra and the staleness metric is plotted as a function of time. Similar anomalies have been observed by Wada et al., Bermbach et al., and Rahman et al., leading to speculation regarding possible causes including side effects of caching and DDoS countermeasures, as well as network jitter [2, 5, 7]. Following the methodology of [5] we run experiments in a single private data center, which isolates the storage system from the effects of caching layers, DDoS countermeasures, and ambient network traffic. None of the mechanisms suggested in prior work account adequately for staleness spikes in such a controlled environment. This work is licensed under the Creative Commons AttributionNonCommercial-NoDerivs 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/. Obtain permission prior to any use beyond those covered by the license. Contact copyright holder by emailing [email protected]. Articles from this volume were invited to present their results at the 41st International Conference on Very Large Data Bases, August 31st September 4th 2015, Kohala Coast, Hawaii. Proceedings of the VLDB Endowment, Vol. 8, No. 7 Copyright 2015 VLDB Endowment 2150-8097/15/03. Our specific contributions with respect to finding the cause of consistency anomalies in Cassandra are as follows: • First, we reproduce the staleness time series experiment of Rahman et al. (see Figure 3 in [5]) in our data center using a more recent version of Cassandra. Our experiments use an improved consistency metric that is more resilient against clock skew, and hence demonstrate that the staleness spikes are not artifacts caused by the use of one specific metric. • Next, we formulate the hypothesis that the observed spikes are caused by processing delays, specifically delays due to the Java virtual machine’s garbage collection (GC) “stop-the-world” (STW) pause. • We test our hypothesis experimentally by plotting both the GC pauses and consistency spikes against time. Our results exhibit strong correlations between the times of occurrence of the GC pauses and consistency spikes, as well as between the durations of the GC pauses and the heights of the spikes. • Guided by insights gained from our experiments we propose a method of removing the staleness spikes by delaying read operations artificially at servers in the time period immediately following a GC pause. This technique yields more than a 98% reduction in the number of consistency anomalies that exceed 5ms, and has negligible impact on throughput and latency.